A method and apparatus for enhancing the broadcast of a live event

ABSTRACT

Pan, tilt and zoom sensors are coupled to a broadcast camera in order to determine the field of view of the broadcast camera and to make a rough estimate of a target&#39;s location in the broadcast camera&#39;s field of view. Pattern recognition techniques can be used to determine the exact location of the target in the broadcast camera&#39;s field of view. If a preselected target is at least partially within the field of view of the broadcast camera, all or part of the target&#39;s image is enhanced. The enhancements include replacing the target image with a second image, overlaying the target image or highlighting the target image. Examples of a target include a billboard, a portion of a playing field or another location at a live event. The enhancements made to the target&#39;s image can be seen by the television viewer but are not visible to persons at the live event.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.09/844,524, filed on Apr. 27, 2001, which is a continuation of U.S.patent application Ser. No. 09/627,106, filed on Jul. 27, 2000, which isa continuation of U.S. patent application Ser. No. 09/264,138, filedMar. 5, 1999, now U.S. Pat. No. 6,141,060, which is a continuation ofU.S. patent application Ser. No. 08/735,020, filed Oct. 22, 1996, nowU.S. Pat. No. 5,917,553, incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention is directed to a method and apparatus forenhancing a television broadcast of a live event.

2. Description of the Related Art

The television presentation of live events could be improved byenhancing the video in real time to make the presentation moreinteresting to the viewer. For example, television viewers cannot seethe entire playing field during a sporting event; therefore, the viewermay lose perspective as to where one of the players or objects are onthe field in relation to the rest of the field, players or objects.During the telecast of football games cameras tend to zoom in on theplayers which allows the viewer to only see a small portion of thefield. Because the viewer can only see a small portion of the field aviewer may not know where a particular player is in relation to thepertinent locations on the field. One instance is when a player iscarrying the football, the television viewer may not know how far thatplayer has to run for a first down. One enhancement that would behelpful to television viewers of football games is to highlight thefield at the point where a player must advance in order to obtain afirst down.

An enhancement that would be helpful to viewers of golf tournaments isto highlight those portions of a golf course that have been notorioustrouble spots to golfers. While the professional golfer is aware ofthese trouble spots and hits the ball to avoid those spots, thetelevision viewer may not be aware of those trouble spots and may wonderwhy a particular golfer is hitting the ball in a certain direction. Ifthe golf course was highlighted to show these trouble spots, atelevision viewer would understand the strategy that the golfer is usingand get more enjoyment out of viewing the golf tournament. Anotheruseful enhancement would include showing the contours of the green.Similar enhancements to the playing field would be useful in othersports as well.

Furthermore, live events do not take advantage of the scope of thetelevision audience with respect to advertising. First, advertisementson display at a stadium can be televised; however, many of thoseadvertisements are not applicable to the television audience. Forexample, a particular sporting event may be played in San Francisco andtelevised around the world. A local store may pay for a billboard at thestadium. However, viewers in other parts of the United States or inother countries receiving the broadcast may not have access to thatstore and, thus, the broadcast of the advertisement is not effective.Second, some of the space at a stadium is not used because such usewould interfere with the view of the players or the spectators at thestadium. However, using that space for advertisement would be veryeffective for the television audience. For example, the glass around theperimeter of a hockey rink would provide an effective place foradvertisements to the television audience. However, such advertisementswould block the view of spectators at the stadium. Third, someadvertisements would be more effective if their exposure is limited toparticular times when customers are thinking of that type of product.For example, an advertisement for an umbrella would be more effectivewhile it was raining.

Previous attempts to enhance the video presentation of live events havenot been satisfactory. Some broadcasters superimpose advertisements onthe screen; however, these advertisements tend to block the view of theevent.

Another solution included digitizing a frame of video and using acomputer with pattern recognition software to locate the target image tobe replaced in the frame of video. When the target image is found, areplacement image is inserted in its place. The problem with thissolution is that the software is too slow and cannot be effectively usedin conjunction with a live event. Such systems are even slower when theyaccount for occlusions. An occlusion is something that blocks thetarget. For example, if the target is a billboard on the boards around ahockey rink, one example of an occlusion is a player standing in frontof the billboard. When that billboard is replaced, the new billboardimage must be inserted into the video such that the player appears to bein front of the replacement billboard.

SUMMARY OF THE INVENTION

The present invention is directed to a system for enhancing thebroadcast of a live event. A target, at a live event, is selected to beenhanced. Examples of targets include advertisements at a stadium,portions of the playing field (e.g., football field, baseball field,soccer field, basketball court, etc.), locations at or near the stadium,or a monochrome background (e.g. for chroma-key) positioned at or nearthe stadium. The system of the present invention, roughly described,captures video using a camera, senses field of view data for thatcamera, determines a position and orientation of a video image of thetarget in the captured video and modifies the captured video byenhancing at least a portion of the video image of the target.Alternative embodiments of the present invention include determining theperspective of the video image of the target and/or preparing anocclusion for the video image of the target.

One embodiment of the present invention includes one or more field ofview sensors coupled to a camera such that the sensors can detect datafrom which the field of view of the camera can be determined. The fieldof view sensors could include pan, tilt and/or zoom sensors. The systemalso includes a processor, a memory and a video modification unit. Thememory stores a location of the target and, optionally, datarepresenting at least a portion of the video image of the target. Theprocessor, which is in communication with the memory and the field ofview sensors, is programmed to determine whether the target is withinthe field of view of the camera and, if so, the position of the targetwithin a frame of video of the camera. Alternate embodiments allow forthe processor to determine the position of the target in the frame ofvideo using field of view data, pattern (or image) recognitiontechnology, electromagnetic signals and/or other appropriate means. Oneexemplar embodiment uses field of view data to find a rough location ofthe target and then uses pattern recognition to find the exact location.Such a combination of field of view data with pattern recognitiontechnology provides for faster resolution of the target's location thanusing pattern recognition alone.

The video modification unit, which is in communication with theprocessor, modifies the frame of video to enhance at least a portion ofthe video image of the target. That is, a target can be edited,highlighted, overlayed or replaced with a replacement image. Forexample, a video modification unit can be used to highlight a portion ofa football field (or other playing field) or replace a first billboardin a stadium with a second billboard. Because the system can beconfigured to use pattern recognition technology and field of viewsensors, the system can be used with multiple broadcast camerassimultaneously. Therefore, a producer of a live event is free to switchbetween the various broadcast cameras at the stadium and the televisionviewer will see the enhancement regardless of which camera is selectedby the producer.

An alternate embodiment contemplates replacing either the field of viewsensors and/or the pattern recognition technology with electromagnetictransmitters and sensors. That is, the target can be used to emit anelectromagnetic signal. A sensor can be placed at the camera, or thecamera can be used as a sensor, to detect the signal from the target inorder to locate the target. Once the target is located within the videoframe, the system can enhance the video image of the target. A furtheralternative includes treating the target with spectral coatings so thatthe target will reflect (or emit) a distinct signal which can bedetected by a camera with a filter or other sensor.

These and other objects and advantages of the invention will appear moreclearly from the following description in which the preferred embodimentof the invention has been set forth in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a perspective view of part of a football stadium.

FIG. 2 depicts a perspective view of the football stadium of FIG. 1 asseen by a television viewer after the video has been enhanced.

FIG. 3 depicts a block diagram of a subset of the components that makeup the present invention.

FIG. 4 depicts a block diagram of a subset of the components that makeup the present invention.

FIG. 5 is a flow chart describing the operation of the presentinvention.

FIG. 6 is a flow chart which provides more detail of how the presentinvention accounts for occlusions.

FIG. 7 is a partial block diagram of an alternate embodiment of thepresent invention.

FIG. 8 is a partial flow chart describing the operation of the alternateembodiment depicted in FIG. 7.

DETAILED DESCRIPTION

FIG. 1 is a partial view of football stadium 100. In the center ofstadium 100 is a football field 102. Surrounding football field 102 arethe seats 104 for the fans. Between seats 104 and playing field 102 is aretaining wall 106. On retaining wall 106 is an advertisement AD1. Forexample purposes only, assume that a particular television broadcasterhas selected four targets for enhancement. The first target is anadvertisement AD 1 to be replaced by another advertisement. The secondtarget is a portion of the playing field which is to receive anadvertisement. For this example, assume that the broadcaster wishes toplace an advertisement in the end zone 108 of the football field. Athird target is an area above the stadium. That is, the televisionbroadcaster may wish that when a camera is pointed to the top of thestadium, the viewers sees an advertisement suspended above the stadium.A fourth target is a location on the playing field 102 representingwhere a team must cross in order to get a first down. Although thetelevision broadcaster may be enhancing the video image as discussedabove, the spectators and players at the stadium would not see any ofthese enhancements, rather they would view the stadium as depicted inFIG. 1.

FIG. 2 shows the view of FIG. 1, as seen by viewers watching thebroadcast on television, after enhancements are made to the video.Advertisement AD2 is in the same location as advertisement AD1 was inFIG. 1. Thus, advertisement AD2 has replaced advertisement AD1.Advertisement AD3 is shown in end zone 108. Advertisement AD3 does notreplace another advertisement because there was no advertisement in endzone 108 prior to the enhancement. FIG. 2 also shows advertisement AD4,which to the television viewer appears to be suspended above stadium100. Also shown in FIG. 2 is a thick line 110 which represents thehighlighting of the portion of the field at which the team who isoffense must cross in order to get a first down at a particular momentduring the game. In this particular example, the highlighting of thefield consists of a bold thick line. Alternatives include differentcolor lines, shading, using a blinking line, varying the brightness,etc. The enhancement need not be a line. The enhancement may also be anyother shape or graphic that is appropriate. Thus, for purposes of thispatent an enhancement includes editing an image, replacing part of animage with another image, overlaying all or part of an image,highlighting an image using any appropriate method of highlighting, orreplacing an image with video.

FIG. 3 is a block diagram of a subset of the components that make up thepresent invention. The components shown on FIG. 3 are typically locatedat a camera bay in the stadium; however, they can be located in othersuitable locations. Broadcast camera 140 captures a frame of video whichis sent to a production center as shown by the signal BC1. Broadcastcamera 140 has a zoom lens, including a 2X Expander (range extender).Connected to broadcast camera 140 is a 2X Expander/zoom/focus sensor 152(collectively a “zoom sensor”) which senses the zoom in the camera, thefocal distance of the camera lense, and whether the 2X Expander is beingused. The analog output of sensor 152 is sent to an analog to digitalconverter 154, which converts the analog signal to a digital signal, andtransmits the digital signal to processor 156. One alternative includesusing a zoom sensor with a digital output, which would remove the needfor analog to digital converter 154. Broadcast camera 140 is mounted ontripod 144 which includes pan and tilt heads that enable broadcastcamera 140 to pan and tilt. Attached to tripod 144 are pan sensor 146and tilt sensor 148, both of which are connected to pan-tilt electronics150. Alternatively, broadcast camera 140 can include a built-in pan andtilt unit. In either configuration, pan sensor 146, tilt sensor 148 andzoom sensor 152 are considered to be coupled to broadcast camera 140because they can sense data representing the pan tilt, and zoom ofbroadcast camera 140.

Processor 156 is an Intel Pentium processor with supporting electronics;however, various other processors can be substituted. Processor 156 alsoincludes memory and a disk drive to store data and software. In additionto being in communication with pan-tilt electronics 150 and analog todigital converter 154, processor 156 is in communication (via signalCB1) with a production center which is described below.

In one embodiment, pan sensor 146 and tilt sensor 148 are opticalencoders that output a signal, measured as a number of clicks,indicating the rotation of a shaft. Forty thousand (40,000) clicksrepresent a full 360° rotation. Thus, a processor can divide the numberof measured clicks by 40,000 and multiply by 360 to determine the pan ortilt angle in degrees. The pan and tilt sensors use standard technologyknown in the art and can be replaced by other suitable pan and tiltsensors known by those skilled in the relevant art. Pan/tilt electronics150 receives the output of pan sensor 146 and tilt sensor 148, convertsthe output to a digital signal (representing pan and tilt) and transmitsthe digital signal to processor 156. The pan, tilt and zoom sensors areused to determine the field of view of the broadcast camera. Thus, oneor more of the pan, tilt or zoom sensors can be labeled as a field ofview senor(s). For example, if a camera cannot zoom or tilt, the fieldof view sensor would only include a pan sensor.

An alternative field of view sensor includes placing marks in variousknown locations in the stadium such that each mark looks different andat least one mark will always be visible to the camera while the camerais pointed at the relevant portions of the stadium. A computer usingpattern recognition technology can find the mark in a frame of videoand, based on the mark's size and position in the frame of video,determine more precisely the field of view and/or pan, tilt or zoom ofthe camera. A system can also be set up to use pan/tilt/zoom sensors incombination with the marks described above so that the pan/tilt/zoom canbe used to make a rough estimate of where the camera is pointing and themark is used to achieve a more accurate estimate. In such a combinationsystem the marks need not look different if the placement of the marksis predetermined. Another alternative includes placing infrared emittersor beacons along the perimeter of the playing field or other portions ofthe stadium. A computer can determine an infrared sensor's field of viewbased on the location of the signal in the infrared sensor's frame ofdata. If the infrared sensor is mounted on a broadcast camera,determining the pan and tilt of the infrared sensor determines the panand tilt of the broadcast camera plus a known offset. A more detaileddiscussion of using infrared technology, pan/tilt/zoom sensors, threedimensional location finding technology and video enhancement can befound in U.S. patent application Ser. No. 08/585,145, A System ForEnhancing The Television Presentation Of An Object At A Sporting Event,incorporated herein by reference.

FIG. 3 shows a second and optional camera labeled as dedicated camera142. Dedicated camera 142 is mounted on a tripod 157. In one embodiment,tripod 157 includes an optional pan sensor 158 and an optional tiltsensor 160, both of which are in communication with pan-tilt electronics150. As will be explained below, in one embodiment the dedicated camerais set to one pan and tilt position; therefore, pan and tilt sensors arenot needed. The output of dedicated camera 142 is the camera signal DC1,which is communicated to the production center described below. Thepresent invention will perform its function without the use of dedicatedcamera 142; however, dedicated camera 142 improves the ability of thesystem to account for occlusions. Dedicated camera 142 should be locatedsubstantially adjacent to broadcast camera 140. That means thatdedicated camera 142 should be as close as possible to broadcast camera140 so that both will function properly yet their optical axes will beas close as practical. Thus, if both cameras are focused on the sameobject, their pan and tilt angle should be very similar. In variousalternatives, each broadcast camera could be associated with more thanone dedicated cameras. In order to further enhance performance, eachbroadcast camera would include a plurality of dedicated cameras, onededicated camera for each potential target the broadcast camera willview.

FIG. 4 is a block diagram of the production center. Typically, theproduction center is housed in a truck parked outside of the stadium.However, the production center can be at a central office or thecomponents of the production center can be spread out in multiplelocations. The heart of the production center is processor 200. Thepreferred processor 200 is an Onyx computer from Silicon Graphics;however, various other suitable processors or combinations of processorscan perform the necessary functions of the present invention. Processor200 is in communication with video control 202, video mixer 204 andmultiplexor 206. In one alternative, processor 200 includes more thanone processor. For example, processor 200 could include two Onyxcomputers, one for locating the target and one for determiningocclusions.

Broadcasters use many broadcast cameras at the stadium to televise asporting event. The video signals from the various cameras are sent tovideo control 202 which is used to select one broadcast camera fortransmission to viewers. One embodiment of video control 202 includes aplurality of monitors (one monitor for each video signal) and aselection circuit. A director (or manager, producer, etc.) can monitorthe different video signals and choose which signals to broadcast. Thechoice would be communicated to the selection circuit which selects onecamera signal to broadcast. The choice is also communicated to processor200, video mixer 204 and multiplexer 206 via signal 208. The selectedvideo signal is sent to delay 210 and processor 200 via analog todigital converter 212. If the broadcast camera is a digital camera, thenthere would be no need for analog to digital converter 212.

The output of delay 210 is sent to video modification unit 214. Thepurpose of delay 210 is to delay the broadcast video signal a fixednumber of frames to allow time for processor 200 to receive data,determine the position of the target in the frame of video and prepareany enhancements. Although the video is delayed a small number offrames, the television signal is still defined as live. The delayintroduced by the system is a small delay (under one second) which doesnot accumulate. That is, different frames of video are enhanced with thesame small delay. For example, a ten frame delay is equivalent toone-third of a second, which is not considered a significant delay fortelevision.

Video mixer 204 receives the video signals from all of the dedicatedcameras. FIG. 4 shows signals DC1 and DC2. Signal DC1 is a dedicatedcamera associated with the broadcast camera BC1. If video control 202selects BC1 then that selection is communicated to video mixer 204 whichselects DC1. As discussed above, it is contemplated that somealternatives include having many dedicated cameras for one broadcastcamera. For example, one broadcast camera may have four dedicatedcameras. In that case, the dedicated cameras would be labeled DC1 a, DC1b, DC1 c and DC1 d. When broadcast camera BC1 is selected, video mixer204 would select up to all four dedicated cameras: DC1 a, DC1 b, DC1 cand DC1 d. The selected signal(s) from video mixer 204 is sent to analogto digital converter 216 which digitizes the video signal(s) and sendsthe digital signal(s) to processor 200.

Multiplexer 206 receives signals from the processors at each of thecamera locations. For example, FIG. 4 shows multiplexer 206 receivingsignal CB1 from processor 156 of FIG. 3. Each of the processor signals(CB1, CB2, . . . ) is associated with a broadcast camera. Thus, theselection by video control 202 is communicated to multiplexer 206 sothat multiplexer 206 can send the corresponding signal to processor 200.The signal sent by multiplexer 206 to processor 200 includes theinformation from the field of view sensors. In one embodiment, processor156 calculates the field of view and sends the resulting information,via multiplexer 206, to processor 200. In another embodiment, processor200 receives the data via multiplexer 206 and determines the field ofview. Either alternative is suitable for the present invention.

Processor 200 is connected to memory 220 which stores the locations ofthe targets and images of the targets (or at least partial images).Memory 220 also stores images of the replacement graphics, instructionsfor creating replacement graphics and/or instructions for highlighting,editing, etc. Memory 200 is loaded with its data and maintained byprocessor 222. The inventors contemplate that during operation of thissystem, processor 200 will be too busy to use compute time for loadingand maintaining memory 220. Thus, a separate processor 222 is used toload and maintain the memory during operation. If cost is a factor,processor 222 can be eliminated and processor 200 will be used to loadand maintain memory 220; however, for optimal performance memory 220should be loaded, if possible, prior to the broadcast.

The images and locations of targets can be loaded into memory 220 eithermanually or automatically. For example, if the target's image andlocation are known in advance (e.g. an advertisement at the stadium)then prior to real-time operation of the system an operator can inputthe location of the target and scan in (or otherwise download) an imageof the target. Alternatively, the operator can point one or more camerasat the target and use a mouse, light pen or other pointing device toselect the target's image for storing in memory 220. The location of thetarget can be determined by physical measurement, using pan/tilt/zoomsensors, etc. If the target is not known in advance (for example if thetarget is the first down yard line) then the operator can select thetarget during operation using a pointing device and the system willdownload the image of the target and its location (using pan/tilt/zoomdata) to memory 220. Alternatively, the system can be programmed to knowthat the target is one of a set of possible targets. For example, thesystem can be programmed to know that the target is a yard line and theoperator need only input which yard line is the current target. Thereplacement graphics are loaded into memory after being digitized,downloaded or the replacement graphics can be created with processor222. Instructions for highlighting or creating replacement graphics canbe programmed using processor 222 or processor 200.

Processor 200 is connected to video modification unit 214. The output ofvideo modification unit 214, labeled as signal 226, is the video signalintended for broadcast. This signal can be directly broadcast or sent toother hardware for further modification or recording. Video modificationunit 214 modifies the video signal from delay 210 with the data/signalfrom processor 200. The type of modification can vary depending on thedesired graphic result. One exemplar implementation uses a linear keyeras a video modification unit 214. When using a keyer, the signal fromthe video processor 200 to the keyer includes two signals: YUV and anexternal key (alpha). The YUV signal is called foreground and the signalfrom delay 210 is called background. Based on the level of the externalkey, the keyer determines how much of the foreground and background tomix to determine the output signal, from 100 percent foreground and zeropercent background to zero percent foreground and 100 percentbackground, on a pixel by pixel basis. Alternatively, video modificationunit 214 can be another processor or video modification unit 214 can bea part of processor 200.

In operation, processor 200 determines the field of view of the selectedbroadcast camera and checks memory 220 to see if any targets are withinthat field of view. If so, processor 200 then determines the exactposition of the target in a frame of video by determining which pixelsrepresent the target. Processor 200 then checks memory 220 for thereplacement graphic or instructions to make a replacement graphic (orhighlight). If the replacement strategy is to highlight a certainportion of a field, then memory 220 may include instructions forchanging the color of a certain portion of the field, shading of acertain portion of the field, etc. Based on the pan, tilt and zoom, andthe actual image of the target, processor 200 determines the size andorientation of the replacement graphic (also called mapping). In oneembodiment, the enhancement includes processor 200 creating a frame ofvideo with a graphic at the position of the enhancement. The framecreated by processor 200 is sent to video modification unit 214 whichcombines the frame from processor 200 with the frame from delay 210. Aswill be described below, processor 200 is also used to account forocclusions. An alternate embodiment includes eliminating the separatevideo modification unit and using processor 200 to edit the video signalfrom the selected broadcast camera.

FIG. 5 is a flow chart which explains the operation of the presentinvention. In step 300, video data is captured by a broadcast camera andis digitized. If the broadcast camera is a digital camera, digitizing isunnecessary. Simultaneously with step 300, pan, tilt and zoom data(field of view data) is sensed in step 302 and the field of view isdetermined in step 304. In step 306, processor 200 determines if any ofthe targets are within the field of view. Memory 200 (depicted in FIG.4) includes a database. In one alternative, the database stores thethree dimensional locations of all the targets. The field of view of abroadcast camera can be thought of as a pyramid whose location anddimensions are determined based on the field of view data. Afterdetermining the dimensions and locations of the pyramid, processor 200accesses memory 220 to determine if any of the targets are within thepyramid. Step 306 is a quick method for determining if there is a targetwithin the field of view of the camera. If not, the process is done andthe system waits until the next frame of data. If there is a targetwithin the field of view of the selected broadcast camera, then theexact position of the target must be determined within the frame ofvideo of the selected broadcast camera.

Preferably, determining the position of the target is a two-stepprocess. In the first step (step 308) a rough estimate is made based onthe pan, tilt and zoom values and in the second step the estimate of thetarget's position is refined (step 310). In regard to step 308, byknowing where the camera is pointed and the target's three dimensionallocation, the target's position in the video frame can be estimated. Theaccuracy of step 308 is determined by the accuracy of the pan/tilt/zoomsensors, the software used to determine the field of view and thestability of the platform on which the camera is located. In somealternatives, the field of view sensor equipment may be so accurate thatthe position of the target is adequately determined and step 310 is notnecessary. In other instances, the pan, tilt and zoom data only providesa rough estimate 308 (e.g a range of positions or general area ofposition) and step 310 is needed to determine a more accurate position.

Step 310 provides a more accurate determination of the target's positionusing pattern recognition techniques which are known in the art. Exampleof known pattern recognition and image processing technology can befound in the following documents: U.S. Pat. No. 3,973,239, PatternPreliminary Processing System; U.S. Pat. No. 4,612,666, AutomaticPattern Recognition Apparatus; U.S. Pat. No. 4,674,125, Real-TimeHierarchal Pyramid Signal Processing Apparatus; U.S. Pat. No. 4,817,171,Pattern Recognition System; U.S. Pat. No. 4,924,507, Real-Time OpticalMultiple Object Recognition and Tracking System and Method; U.S. Pat.No. 4,950,050, Optical Target Recognition System; U.S. Pat. No.4,995,090, Optoelectronic Pattern Comparison System; U.S. Pat. No.5,060,282, Optical Pattern Recognition Architecture Implementing TheMean-Square Error Correlation Algorithm; U.S. Pat. No. 5,142,590,Pattern Recognition System; U.S. Pat. No. 5,241,616, Optical PatternRecognition System Utilizing Resonator Array; U.S. Pat. No. 5,274,716,Optical Pattern Recognition Apparatus; U.S. Pat. No. 5,465,308, PatternRecognition System; U.S. Pat. No. 5,469,512, Pattern Recognition Device;and U.S. Pat. No. 5,524,065, Method and Apparatus For PatternRecognition. It is contemplated that step 310 can use suitabletechnology other than pattern recognition technology.

In step 312, processor 200 fetches the replacement graphic from memory220. If memory 220 is storing instructions for replacement graphics,then processor 200 fetches the instructions and creates the graphic. Forexample, creating the graphic can include drawing a highlight for theyard line of a football field. In step 314, processor 200 determines thesize and orientation of the replacement image, and maps the replacementimage to the video frame. Memory 220 merely stores one size image.Because of the pan, tilt and zoom of the broadcast camera, the imagestored in memory 220 may need to be mapped to the video frame (e.g.magnified, reduced, twisted, angled, etc.). Processor 200 can determinethe orientation based on the field of view data and/or the patternrecognition analysis in step 310. For example, by knowing where thebroadcast camera is located and the pan, tilt and zoom of the broadcastcamera, a computer can be programmed to figure how to map thereplacement image or highlight on to the video frame.

In step 316, the system accounts for occlusions. If there is an objector person in front of the target, then the enhanced video should showthe object or person in front of the replacement graphic, highlight,etc. In one embodiment, the system cuts out a silhouette in the shape ofthe object or person from the replacement image. Step 316 is discussedin more detail with respect to FIG. 6.

In step 318, the system modifies the video of the original broadcastcamera. As discussed above, this could include creating a second frameof video which includes a replacement image and using a keyer to combinethe second frame of video with the original frame of video.Alternatively, a processor can be used to edit the frame of video of thebroadcast camera. It is possible that within a given frame of videothere may be more than one target. In that case steps 308-318 may berepeated for each target, or steps 308-316 may be repeated for eachtarget and step 318 be performed only once for all targets. Subsequentto step 318, the enhanced frame of video may be broadcast or stored, andthe process (steps 300-318) may repeat for another frame of video.

FIG. 6 is a more detailed flow diagram explaining how the systemaccounts for occlusion. The steps described in FIG. 6 are performed by asystem which includes one or more dedicated cameras (e.g. dedicatedcamera 142). Step 350, is performed before the live event occurs. In oneembodiment, there is a dedicated camera substantially adjacent to abroadcast camera for each target that the broadcast camera may view. Forexample, if there are three advertisements which are to be replaced inthe stadium and a particular camera can view two of thoseadvertisements, then the system can include two dedicated camerassubstantially adjacent to that particular camera. Prior to the game, adedicated camera is pointed directly at one of the targets; the camerais zoomed in such that the target fills a substantial portion of thededicated camera's frame of video; and the image of the target is storedin memory 220. A substantial portion means that the target typicallyappears to cover over half of the frame of video of the dedicatedcamera. For optimal results, the dedicated camera should be zoomed insuch that the target fills the greatest amount of the frame of videopossible while remaining completely within the frame of video, unless itis desired to have clues of the scenery surrounding the target. Afterthe dedicated camera is pointed at the target, its pan, tilt and zoomshould remain fixed.

Once the television broadcast of the live event begins, steps 352-362are repeated for each frame where the occlusion analysis is desired. Instep 352, a video image is captured and digitized by the dedicatedcamera. Simultaneously, a video image is captured by the broadcastcamera. In step 354, the digitized image from the dedicated camera iscompared to the stored image of the target. The stored image is storedin memory 220. The processor knows which stored image to compare withfrom step 306 of FIG. 5. The step of comparing could include alteringone of the images such that both images are the same size andorientation, and then subtracting the data. Alternatively, other methodscan be used to compare. If there is an occlusion blocking the target(step 356), then the two images will be significantly different and, instep 358, an occlusion will be reported. In reporting the occlusion, thesystem reports the presence of an occlusion and the coordinates of theocclusion. When performing step 354, it is possible that there is noocclusion; however, the two images are not exactly the same. Thedifferences between the images must meet a certain minimum threshold tobe considered an occlusion. If the differences are not great enough tobe an occlusion, then in step 360 the system determines that thedifferences are due to ambient conditions in the stadium. For example,if the lights have been dimmed then the captured image of the target mayappear darker. Weather conditions could also have an effect on theappearance of the target image. If small differences are detected instep 360 that do not meet the threshold for occlusions, then the system“learns” the changes to the target by updating the stored image of thetarget to reflect the new lighting or weather conditions (step 362). Forexample, the new stored image of the target may be darker than theoriginal image. Subsequent to step 362 the system performs the reportstep 358 and reports that no occlusion was found.

An alternative to the method of FIG. 6 includes comparing the targetimage from the broadcast camera to the stored image. However, using thebroadcast camera is not as advantageous as using a dedicated camerabecause it is likely that the broadcast camera would not be zoomed tothe image. Thus, the target image is likely to be smaller on thebroadcast camera than it will on the dedicated camera. Because there isa small image to work with, the system loses the subpixel accuracyobtained from the dedicated camera. Also, using a separate dedicatedcamera may increase the speed at which the system accounts forocclusions.

FIG. 7 shows an alternative embodiment of the present invention whichutilizes electromagnetic transmitting beacons at or near a target. Thebeacons transmit an electromagnetic signal not visible to the human eye.Electromagnetic waves include light, radio, x-rays, gamma rays,microwave, infrared, ultraviolet and others, all involving thepropagation of electric and magnetic fields through space. Thedifference between the various types of electromagnetic waves are in thefrequency or wave length. The human eye is sensitive to electromagneticradiation of wave lengths from approximately 400-700 nm, the rangecalled light, visible light or the visible spectrum. Thus, the phrase“electromagnetic signal not visible to a human eye” means anelectromagnetic wave outside of the visible spectrum. It is importantthat the signal transmitted from the beacon is not visible to human eyeso that the visual appearance of the target will not be altered to thosepeople attending the live event. In one embodiment, the beacon is anelectromagnetic transmitter which includes infrared emitting diodes.Other sources which transmit electromagnetic waves may also used, forexample, radio transmitters, radar repeaters, etc.

FIG. 7 shows a broadcast camera 400 which outputs a video signal 402.Broadcast camera 400 includes a zoom lens coupled to a zoom detector404. The output of zoom detector 404 is transmitted to analog to digitalconverter 406 which sends the digital output to processor 408. Mountedon top of broadcast camera 400 is sensor 410. In the embodiment whichutilizes an infrared emitter as a beacon, sensor 410 is an infraredsensor. Sensor 410 is mounted on top of broadcast camera 400 so that theoptical axis of sensor 410 is as close as possible to the optical axisof broadcast camera 400. It is also possible to locate sensor 410 nearbroadcast camera 400 and account for differences between optical axesusing matrix transformations or other suitable mathematics.

One example of an infrared sensor is a progressive scan, full frameshutter camera, for example, the TM-9701 by Pulnix. The Pulnix sensor isa high resolution 768(H) by 484(V) black and white full frame shuttercamera with asynchronous reset capability. The camera has an eight bitdigital signal output and progressively scans 525 lines of video data. Anarrow band infrared filter is affixed in front of the lens of thePulnix sensor. The purpose of the filter is to block electromagneticsignals that are outside the spectrum of the signal from the beacon. Thesensor captures a frame of video (data) which comprises a set of pixels.Each pixel is assigned a coordinate corresponding to an x-axis and ay-axis. The sensor data includes an eight bit brightness value for eachpixel, which are scanned out pixel by pixel to interface 412 along withother timing information. Interface 412 outputs four signals: LDV, FDV,CK and DATA. LDV (line data valid) is transmitted to X-Y counters 414and indicates that a new line of valid data is being scanned out ofsensor 410. FDV (frame data valid) which is transmitted to X-Y counters414 and memory control 416, indicates that valid data for the next frameis being transmitted. CK (pixel clock) is a 14.318 MHZ clock from sensor414 sent to X-Y counters 414 and memory control 416. X-Y counters 414counts X and Y coordinates sequentially in order to keep track of thelocation of the pixel whose data is being scanned in at the currenttime. When LDV is inserted, the X counter is reset. When FDV isinserted, the Y counter is reset.

The signal Data includes the eight bit data value for each pixel. Asdata is read from sensor 410, memory control 416 determines whether thepixels meets a brightness threshold. That is, noise and other sourceswill cause a large number of pixels to receive some data. However, thepixels receiving the signal from the beacon will have at least a minimumbrightness level. This brightness threshold is set in a register (notshown) which can be set by processor 408. If the data for a particularpixel is above the brightness threshold, memory control 416 sends awrite enable (WE) signal to memory 418, causing memory 418 to store theX and Y coordinates of the pixel, the data for that pixel and a code forthat pixel. The code indicates that the data is valid data, a new frame,end of frame or a flash. Processor 408 can read the data from memory 418and process the data locally or transmit the data to the productioncenter (e.g., to multiplexer 206).

Many arenas do not allow photographers to use flashes on their camerasin order to prevent impairing a player's vision from random flashesduring a sporting event. In lieu of individual camera flashes, manyarenas install a set of strobe flashes at or near the ceiling of thearenas and provide for communication between each photographer's cameraand the set of strobe flashes. When the photographer takes a picture,the strobe flashes emit a flash of light, which may include anelectromagnetic wave in the infrared spectrum. In one embodiment, thesystem avoids using incorrect data due to sensors detecting a flash byusing filters. A second embodiment connects a signal from a strobe flashto a computer which causes the system to ignore data sensed during aflash. A third embodiment includes using flash detectors. The flashdetector can be located anywhere in the arena suitable for sensing astrobe flash. FIG. 7 shows flash detector 422 which detects a flash andsends a signal to memory control 416. Flash detector 422 includes aphoto detector which can comprise, at least, a photo diode and an opamp.In front of the photo detector would be a filter that allows detectionof signals in a spectrum that includes the signals emitted by thebeacon. Connected to the opamp are components which can detect pulseedges.

The embodiment described in FIG. 7 operates similar to the embodimentdescribed in FIG. 3. Some of the differences between the operation ofthe two embodiments are depicted in FIG. 8. Similar to the embodiment inFIG. 3, the embodiment in FIG. 7 first captures and digitizes videodata. In step 450, infrared data is received. In step 452, the systemdetermines whether a target is found in the infrared data by monitoringthe data stored in memory 418. Since memory control 416 only allows dataabove a threshold to be stored in memory 418, if a given frame of datafrom a sensor has pixel data stored in memory then a target is found. Ifa sensor is detecting false targets, then various error correctionmethods known in the art can be utilized. In step 454, the position ofthe target is determined in the frame of video by reading the X and Ycoordinates stored with the pixel data in memory 418. Step 456 finetunes the determined position information of the target to account forthe error from the camera's platform or pan/tilt/zoom sensors. Onealternative for accounting for the difference in optical axis is to usea transformation matrix; however, other mathematical solutions known inthe art are also suitable. After step 456, the system can perform steps312 through 318 as described with respect to FIG. 5, however, any fieldof view data used is based on the size and position of the beacon'ssignal in the sensor's frame of video.

A further alternative of FIG. 7 includes using polarization. That is theinfrared filter on sensor 410 is replaced or augmented with a polarizedfilter. A target to be replaced (e.g., a billboard) is treated with aspectral coating that allows only polarized light to reflect off thebillboard. The filter and spectral coating are designed such that lightreflecting off the billboard to sensor 410 will be completelyblacked-out. The pixels that represent the position of the target in thesensor's frame of video will have a brightness value of zero or close tozero. Thus, memory control 416 is used to only store memory that has abrightness value of zero or below a threshold level.

The foregoing detailed description of the invention has been presentedfor purposes of illustration and description. It is not intended to beexhaustive or to limit the invention to the precise form disclosed, andobviously many modifications and variations are possible in light of theabove teaching. The described embodiments of the system for enhancingthe broadcast of a live event were chosen in order to best explain theprinciples of the invention and its practical application to therebyenable others skilled in the art to best utilize the invention invarious embodiments and with various modifications as are suited to theparticular use contemplated. The invention is, thus, intended to be usedwith many different types of live events including various sportingevents and nonsporting events. It is intended that the scope of theinvention be defined by the claims appended hereto.

1. A method for enhancing the broadcast of a live event, comprising thesteps of: capturing first video using a first camera; sensing field ofview data representing a field of view of said first camera; determininga position and orientation of a video image of a target in said capturedvideo at least partially based on recognizing one or more portions ofsaid video image of said target in said captured video; and modifyingsaid captured video data by enhancing at least a segment of said videoimage of said target.